Control apparatus and control method for determining relation of persons included in an image, and storage medium storing a program therefor

ABSTRACT

A control apparatus includes a detection unit, an association unit, and an output control unit. The detection unit detects a person from an image which includes a plurality of persons. The association unit associates the persons included in the image with each other based on at least one of a position of the person detected by the detection unit, directions of faces of the persons included in the image, and distances between the persons included in the image. The output control unit causes an output unit to output information that is indicative of a relation of the detected person with respect to other persons included in the image based on a result of association performed by the association unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation, and claims the benefit, of U.S.patent application Ser. No. 15/334,990 filed Oct. 26, 2016, which claimsthe benefit of Japanese Patent Application Numbers 2015-214925 filedOct. 30, 2015, 2015-214935 filed Oct. 30, 2015, and 2015-215053 filedOct. 30, 2015, all of which are hereby incorporated by reference hereinin their entirety.

BACKGROUND OF THE INVENTION Field of Invention

Aspects of the present invention generally relate to a controlapparatus, a control method, and a storage medium for detecting a personfrom an image and causing a display device to perform displayprocessing.

Description of Related Art

Image processing techniques for detecting a person from an image arewell known. Japanese Patent Application Laid-Open No. 2014-199506discusses a method for detecting a shielded person from an image.

The technique discussed in Japanese Patent Application Laid-Open No.2014-199506 is able to indicate the position of a person in an image.For example, the technique is able to surround a person present in animage with a circle and to display the position of the person to a user.However, since this technique indicates only the position of a person,the user cannot determine at a glance whether the detected person isindependently working in isolation from a group or is workingcollaboratively near the other persons. For example, in the case of asite in which data to be prevented from leakage is treated, a rule thatprohibits working independently may be imposed, and a supervisor mayobserve workers via an imaging system and may want to call attention toa worker who is independently working in isolation from a group.Alternatively, in situations where a rule for working independently isenforced, for example, in a test taking environment, a supervisorobserving a group of persons via the imaging system may desire to callattention to a person who is interacting with or observing other membersof the group. However, only viewing a captured image is not enough forthe supervisor to determine whether each person shown in the capturedimage is working independently or is working in collaboration with aplurality of persons existing in an image.

SUMMARY OF THE INVENTION

According to at least one embodiment disclosed herein, a controlapparatus includes a detection unit configured to detect a person froman image which includes a plurality of persons, an association unitconfigured to associate the persons included in the image with eachother based on at least one of a position of the person detected by thedetection unit, directions of faces of the persons included in theimage, and distances between the persons included in the image, and anoutput control unit configured to cause an output unit to outputinformation that is indicative of a relation of the detected person withrespect to other persons included in the image based on a result ofassociation performed by the association unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates examples of a system configuration and a hardwareconfiguration of a system.

FIG. 2 illustrates examples of functional configurations of an imagingapparatus and a client apparatus.

FIG. 3 is a flowchart illustrating an example of processing for displaycontrol.

FIG. 4 illustrates an example of a captured image.

FIGS. 5A and 5B illustrate examples of images representing a result ofperson detection.

FIG. 6 illustrates an example of a horizontal plane (top view) image.

FIG. 7 illustrates an example of a captured image in which patterns aresuperimposed.

FIG. 8 illustrates an example of a horizontal plane (top view) image inwhich patterns are superimposed.

FIG. 9 illustrates an example of a captured image in which detectionareas are set.

FIG. 10 illustrates an example of a horizontal plane (top view) image inwhich detection areas are set.

FIG. 11 is a flowchart illustrating an example of processing fornotification control.

FIG. 12 is a top view diagram used to illustrate processing fordetermining whether a target person is an isolated from other personsexisting in the image.

FIG. 13 is a flowchart illustrating an example of processing fornotification control.

FIGS. 14A and 14B illustrate examples of work area frames.

FIGS. 15A and 15B illustrate examples of display screens observed by auser.

FIGS. 16A and 16B illustrate further examples of display screensobserved by a user.

FIGS. 17A and 17B illustrate examples of display screens.

FIGS. 18A and 18B illustrate examples of cooperation with an entranceand exit management system.

FIG. 19 is a flowchart illustrating an example of processing fornotification control.

FIG. 20 illustrates an example of determination.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

In a first exemplary embodiment, a description is made about processingfor displaying to a user (a work observer, supervisor or manager) anotification indicating whether each worker shown in a captured image isan isolated worker (a person who is not associated with any otherperson) or a collaborative worker (a person who is associated withanother person) at a site in which data to be prevented from leakage istreated and workers are required to work collaboratively in groups oftwo or more workers. Such a situation is an example of a situation towhich the present exemplary embodiment is applicable, and the presentexemplary embodiment is also applicable to various situations in whichit is required to determine whether a person is present in isolation ora plurality of persons is present collectively. For example, the presentinvention can be applied to a situation in which determining whether,for example, passersby or visitors are alone or a group is utilized tosolve a social issue or work task.

FIG. 1 illustrates examples of a system configuration and a hardwareconfiguration of a system according to the present exemplary embodiment.Referring to FIG. 1, an imaging apparatus 110 performs image capturing.A client apparatus 120 drives the imaging apparatus 110, and displays acaptured image, which is captured by the imaging apparatus 110. An inputdevice 130, which includes, for example, a mouse and a keyboard, is usedto input user instructions to the client apparatus 120. A display device140, which includes, for example, a display, displays an image outputfrom the client apparatus 120. While, in FIG. 1, the client apparatus120 and the display device 140 are illustrated as respective independentapparatuses, the client apparatus 120 and the display device 140 can beconfigured into an integrated apparatus. Moreover, the input device 130and the display device 140 can be configured into an integrated device,and the client apparatus 120, the input device 130, and the displaydevice 140 can be configured into an integrated apparatus. A network 150interconnects the imaging apparatus 110 and the client apparatus 120.The network 150 includes, for example, a plurality of routers, switches,and cables conforming to a communication standard, such as a local areanetwork. In the present exemplary embodiment, the communicationstandard, scale, and configuration of the network 150 are not consideredas long as it allows communication between the imaging apparatus 110 andthe client apparatus 120. For example, the network 150 can include, forexample, the Internet, a wired local area network (LAN), a wireless LAN,or a wide area network (WAN). Additionally, the number of imagingapparatuses connected to the client apparatus 120 is not limited to one,but can be plural.

A hardware configuration of the imaging apparatus 110 is described. Thehardware configuration of the imaging apparatus 110 includes an imagingunit 11, a central processing unit (CPU) 12, a memory 13, and acommunication interface (I/F) 14. The imaging unit 11 includes an imagesensor and an optical system for an object, which is located on theimage sensor, and performs image capturing on the image sensor with anintersection between the optical axis of the optical system and theimage sensor as an imaging center under the control of the CPU 12. Theimage sensor is, for example, a complementary metal-oxide semiconductor(CMOS) sensor or a charge coupled device (CCD) sensor. The CPU 12controls the entire imaging apparatus 110. The memory 13 storesprograms, images captured by the imaging unit 11, and, for example, dataused for the CPU 12 to perform processing. The communication I/F 14controls communication with the client apparatus 120 via the network 150under the control of the CPU 12. The functions of the imaging apparatus110 are implemented by the CPU 12 performing processing based on theprograms stored in the memory 13. Additionally, a processor other thanthe CPU 12 can be used.

A hardware configuration of the client apparatus 120 is described. Theclient apparatus 120 includes a memory 21, a CPU 22, a communication I/F23, an input I/F 24, and a display I/F 25. The CPU 22 controls theentire client apparatus 120. The memory 21 stores programs, capturedimages sent from the imaging apparatus 110, and, for example, data usedfor the CPU 22 to perform processing. The communication I/F 23 controlscommunication with the imaging apparatus 110 via the network 150 basedon the control of the CPU 22. The input I/F 24 interconnects the clientapparatus 120 and the input device 130, and controls inputting ofinformation from the input device 130. The display I/F 25 interconnectsthe client apparatus 120 and the display device 140, and controlsoutputting of information to the display device 140. The functions ofthe client apparatus 120 and processing operations in the flowchart ofFIG. 3, which is described below, are implemented by the CPU 22performing processing based on the programs stored in the memory 21.However, the hardware configuration of the client apparatus 120 is notlimited to that illustrated in FIG. 1. For example, the client apparatus120 can include an audio output device, such as a speaker, which is anexample of a notification device. In a case where the client apparatus120 includes an audio output device as a hardware configuration, anexample of notification processing is processing for outputting a sound,such as an alarm.

FIG. 2 illustrates examples of functional configurations of the imagingapparatus 110 and the client apparatus 120. The functional configurationof the imaging apparatus 110 includes a control unit 111, a signalprocessing unit 112, a drive control unit 113, and a communicationcontrol unit 114. The control unit 111 controls the imaging apparatus110. The signal processing unit 112 performs processing on an imagesignal obtained by image capturing by the imaging unit 11. For example,the signal processing unit 112 performs coding of an image captured bythe imaging unit 11. The signal processing unit 112 is able to use, forexample, Joint Photographic Experts Group (JPEG) as a coding method.Alternatively, the signal processing unit 112 is able to useH.264/MPEG-4 AVC (hereinafter referred to as “H.264”) as a codingmethod. More alternatively, the signal processing unit 112 is able touse High Efficiency Video Coding (HEVC) as a coding method. However, thecoding method is not limited to those. Additionally, the signalprocessing unit 112 can select a coding method from among a plurality ofcoding methods to perform coding.

The drive control unit 113 performs control to change the imagingdirection and angle of view of the imaging unit 11. While, in thepresent exemplary embodiment, a case is described in which the imagingunit is able to change the imaging direction in panning directions andtilting directions and to change the imaging angle of view, this is notlimiting. The imaging apparatus 110 can be configured not to include thefunction of changing the imaging direction in panning directions andtilting directions, or not to include the function of changing the angleof view. The communication control unit 114 sends the captured imagesubjected to processing by the signal processing unit 112 to the clientapparatus 120. Moreover, the communication control unit 114 receives acontrol instruction for the imaging apparatus 110 from the clientapparatus 120.

The functional configuration of the client apparatus 120 includes acontrol unit 121, a communication control unit 123, an informationacquisition unit 124, and a display control unit 125. The control unit121 controls the client apparatus 120. The communication control unit123 receives a captured image from the imaging apparatus 110. Theinformation acquisition unit 124 receives user instructions input viathe input device 130, and acquires input information from the inputdevice 130. The display control unit 125 outputs a video to the displaydevice 140, thus causing the display device 140 to perform displayprocessing, which is described below. The display device 140 is anexample of a notification device. Additionally, the display processingis an example of notification processing.

Next, the flow of processing for display control in the presentexemplary embodiment is described with reference to FIG. 3. FIG. 3 is aflowchart illustrating an example of processing for display control inthe client apparatus 120. The client apparatus 120 acquires a capturedimage (or a moving image) from the imaging apparatus 110, superimposes apattern for an isolated worker or a pattern for a collaborative workeron a worker shown in the acquired captured image, and displays thesuperimposed image on the display device 140.

In step S200, the control unit 121 acquires a captured image from theimaging apparatus 110. FIG. 4 illustrates an example of a captured image300 acquired by the client apparatus 120 from the imaging apparatus 110.In FIG. 4, workers 301 and 302 and workers 303 and 304 representcollaborative workers who are working in pairs. On the other hand, aworker 305 represents an isolated worker who is working alone. Moreover,a work table 306 represents a work table located far away from theimaging apparatus 110. A work table 307 represents a work table locatednear the imaging apparatus 110.

At this time, it is desirable that the position, mounting angle, andangle of view of the imaging unit 11 of the imaging apparatus 110 beadjusted in advance in such a manner that workers do not overlap as muchas possible in a captured image acquired by the client apparatus 120. Inthe present exemplary embodiment, the client apparatus 120 performsprocessing on the captured image acquired from the imaging apparatus110. With this, processing can be performed on a real-time live video.However, this is not limiting because image processing can also beperformed in prerecorded video. For example, the client apparatus 120can acquire a moving image stored in the imaging apparatus 110 andperform processing on each frame, or can perform processing on eachframe of a moving image stored in the client apparatus 120.Alternatively, the client apparatus 120 can access, for example, arecording server and perform processing on each frame of a moving imagestored in the recording server. Moreover, while, in the presentexemplary embodiment, collaborative workers are paired, this is notlimiting. The client apparatus 120 can set three or more workers as agroup of collaborative workers.

Then, in step S201, the control unit 121 performs person detectionprocessing on the captured image 300 acquired in step S200. To performthe person detection processing, in the present exemplary embodiment,the control unit 121 first performs scaling on the captured image 300 invarious sizes. Performing scaling enables detecting persons of varioussizes. Next, the control unit 121 performs raster scan on each scaledcaptured image using a detection window of a specified size. At thistime, the control unit 121 previously calculates a feature amount of aperson using learning data, and determines that the object is a personin a case where a difference between the feature amount calculatedwithin the detection window during scan and the feature amount that isbased on the learning data is smaller than a threshold value.

FIGS. 5A and 5B illustrate examples of images 400 representing a resultof performing person detection on the captured image 300. In FIG. 5A,ellipses 401 to 405 represent positions where the workers 301 to 305have been detected respectively. Furthermore, arrows 406 to 408, whichare half lines with the arrow tail end serving as a base point and thearrow head extending infinitely, represent direction of faces of therespective persons. Furthermore, besides the above-described detectionmethod using the feature amount of a person, for example, a rule thatobliges workers to wear at least one of predetermined uniforms, bibs,and caps, as illustrated in FIG. 5B, can be determined. Morespecifically, the feature amount of a thing worn by a person to bedetected as a worker can be stored in advance as a setting file for arule in, for example, the memory 21 of the client apparatus 120. Indoing so, the control unit 121 can detect a worker wearing, for example,a uniform matching the rule based on the rule in the setting file andthe feature amount of, for example, color and shape of a thing worn by aperson targeted for processing in the captured image.

Then, in step S202, the control unit 121 converts the position of theperson detected in step S201 into coordinates on a horizontal planeimage, which is obtained by projecting a shooting range of the capturedimage onto a horizontal plane from directly above. In the presentexemplary embodiment, it is determined in advance which points on thehorizontal plane image are associated with the respective points on thecaptured image. For example, the user specifies four points on thecaptured image displayed on, for example, the display device 140 andfour points on the horizontal plane image by operating, for example, theinput device 130. The control unit 121 calculates a nomography matrix Hin advance based on, for example, the four points on the captured imageand the four points on the horizontal plane image, which are input bythe user, and the following formulae (1) and (2):

$\begin{matrix}{H = \begin{bmatrix}h_{11} & h_{12} & h_{13} \\h_{21} & h_{22} & h_{23} \\h_{31} & h_{32} & 1\end{bmatrix}} & (1) \\{\begin{bmatrix}h_{11} \\h_{12} \\h_{13} \\h_{21} \\h_{22} \\h_{23} \\h_{31} \\h_{32}\end{bmatrix} = {\begin{bmatrix}p_{x_{1}} & p_{y_{1}} & 1 & 0 & 0 & 0 & {{- p_{x_{1}}}q_{x_{1\;}}} & {{- p_{y_{1}}}q_{x_{1}}} \\0 & 0 & 0 & p_{x_{1}} & p_{y_{1}} & 1 & {{- p_{x_{1}}}q_{y_{1}}} & {{- p_{y_{1}}}q_{y_{1}}} \\p_{x_{2}} & p_{y_{2}} & 1 & 0 & 0 & 0 & {{- p_{x_{2}}}p_{x_{2}}} & {{- p_{y_{2}}}q_{x_{2}}} \\0 & 0 & 0 & p_{x_{2}} & p_{y_{2}} & 1 & {{- p_{x_{2}}}q_{y_{2}}} & {{- p_{y_{2}}}q_{y_{2}}} \\p_{x_{3}} & p_{y_{3}} & 1 & 0 & 0 & 0 & {{- p_{x_{3}}}q_{x_{3}}} & {{- p_{y_{3}}}q_{x_{3}}} \\0 & 0 & 0 & p_{x_{3}} & p_{y_{3}} & 1 & {{- p_{x_{3}}}q_{y_{3}}} & {{- p_{y_{3}}}q_{y_{3}}} \\p_{x_{4}} & p_{y_{4}} & 1 & 0 & 0 & 0 & {{- p_{x_{4}}}p_{x_{4}}} & {{- p_{y_{4}}}q_{x_{4}}} \\0 & 0 & 0 & p_{x_{4}} & p_{y_{4}} & 1 & {{- p_{x_{4}}}q_{y_{4}}} & {{- p_{y_{4}}}q_{y_{4}}}\end{bmatrix}^{- 1}\begin{bmatrix}q_{x_{1}} \\q_{y_{1}} \\q_{x_{2}} \\q_{y_{2}} \\q_{x_{3}} \\q_{y_{3}} \\q_{x_{4}} \\q_{y_{4}}\end{bmatrix}}} & (2)\end{matrix}$

Here, px1, py1, . . . , px4, py4 and qx1, qy1, . . . , qx4, qy4 informula (2) denote x and y coordinates of the four points on thecaptured image and x and y coordinates of the four points on thehorizontal plane image, which are specified by the user.

Next, the control unit 121 applies the following formula (3) to the xcoordinate px and the y coordinate py on the captured image and thenomography matrix H, thus calculating the X coordinate qx and the Ycoordinate qy on the horizontal plane image corresponding to thecoordinates px and py.

$\begin{matrix}{\begin{bmatrix}{sq}_{x} \\{sq}_{y} \\s\end{bmatrix} = {H\begin{bmatrix}p_{x} \\p_{y} \\1\end{bmatrix}}} & (3)\end{matrix}$

FIG. 6 illustrates an example of a horizontal plane image 500 at a worksite. In FIG. 6, coordinates 501 to 505 represent a result of convertingrepresentative points of the ellipses 401 to 405, which are respectiveperson detection positions, into coordinates on the horizontal planeimage. In the present exemplary embodiment, the representative point issupposed to be the center of a person detection position as an example.The representative point can be the center of a region corresponding tothe head of a person. Moreover, regions 506 and 507 represent regionsobtained when the work tables 306 and 307 are expressed on thehorizontal plane image, respectively. While, in the present exemplaryembodiment, a sketch of the work site made by, for example,computer-aided design (CAD) is used as the horizontal plane image, thisis not limiting. For example, an image obtained by deforming thecaptured image can also be used, or a handwritten image can also beused. Moreover, an image does not need to be prepared, and onlyinformation about the above-mentioned coordinates can be prepared insidea program. In other words, the horizontal plane image 500 does not needto be generated, and only calculating the coordinates on the horizontalplane image 500 can be performed. Furthermore, if image capturing isperformed with a composition close to the horizontal plane image 500(for example, an overhead view), processing in step S203 does not needto be necessarily performed. In this case, distances on the capturedimage can be used to determine whether the target worker is an isolatedworker or a collaborative worker. Additionally, the method of generatingthe horizontal plane image 500 can include other various methods.

Next, in step S203, the control unit 121 generates a list of detectedpersons. At this time, the control unit 121 stores coordinates ofrepresentative points of the detected persons in the horizontal planeimage at the respective nodes in the list.

Next, in steps S204 to S209, the control unit 121, while scanning thegenerated list, determines whether a person of interest is an isolatedworker or a collaborative worker, and superimposes a patternrepresenting an isolated worker or a pattern representing acollaborative worker on every person shown in the captured image. Theflow of processing is as follows.

(Processing 1). In step S204, the control unit 121 sets a person setforth at the head of the generated list as a processing target.

(Processing 2). In step S205, the control unit 121 determines whether aperson of interest is an isolated worker.

In the present exemplary embodiment, the control unit 121 sets thecoordinates of the person of interest on the horizontal plane image asthe center, and determines whether a detected person is present withinthe radius R from the center. If a detected person is present within theradius R, the control unit 121 determines that the person of interest isa collaborative worker (NO in step S205), and then proceeds to(Processing 3-1) (step S206). On the other hand, if no detected personis present within the radius R, the control unit 121 determines that theperson of interest is an isolated worker (YES in step S205), and thenproceeds to (Processing 3-2) (step S207). In other words, in a casewhere a detected person is present within the radius R from the centercorresponding to the coordinates of a person of interest on thehorizontal plane image, the control unit 121 determines that the personof interest and the detected person are associated with each other and,thus, that the person of interest is a collaborative worker. In thisway, the control unit 121 functions as an association unit. Moreover,the radius R is a predetermined setting value. The radius R can be setas appropriate according to a situation to which the present exemplaryembodiment is applied. For example, in the case of observing an indoorwork, the radius R can be set to a distance corresponding to 2 m.

(Processing 3-1). In step S206, the control unit 121 superimposes apattern representing a collaborative worker on the person of interestand the person present within the radius R from the person of intereston the captured image and the horizontal plane image.

(Processing 3-2). In step S207, the control unit 121 superimposes apattern representing an isolated worker on the person of interest on thecaptured image and the horizontal plane image.

(Processing 4). In step S208, the control unit 121 determines whether aperson set forth at the last of the list is set as a processing target.

If a person set forth at the last of the list is not set as a processingtarget (NO in step S208), then in step S209, the control unit 121 setsthe next person as a processing target, and then returns to (Processing2) (step S205). On the other hand, if a person set forth at the last ofthe list is set as a processing target (YES in step S208), the controlunit 121 ends processing for superimposing a pattern representing anisolated worker or a pattern representing a collaborative worker onevery person, and then proceeds to step S210.

Here, processing for determining whether the person of interest is anisolated worker or a collaborative worker is not limited to theabove-described processing. For example, the control unit 121 can beconfigured to associate persons with each other and determine that theassociated person is a collaborative worker based on at least one ofpositions of persons detected from the captured image, directions offaces of the persons, and distances between the persons. For example,the control unit 121 can be configured to associate persons presentwithin a predetermined range from a predetermined object, such as a worktable, in the captured image with each other, or can be configured toestimate directions (or eye lines) of faces of persons and to associatepersons the directions of faces of whom intersect with each other.Moreover, for example, the control unit 121 can be configured toassociate persons the distance between whom is within a predetermineddistance with each other. Then, the control unit 121 can be configuredto determine that persons each of whom is associated with another personare collaborative workers and a person who is not associated with anyother person is an isolated worker. Additionally, the control unit 121can be further configured to associate persons with each other based oncolors or shapes of uniforms, bibs (vest-like clothes), or caps ofpersons. For example, the control unit 121 can be configured toassociate persons the shapes of uniforms of whom are the same amongpersons the distance between whom is within a predetermined distancewith each other, and not to associate persons the shapes of uniforms ofwhom are different even among persons the distance between whom iswithin a predetermined distance with each other.

FIG. 7 illustrates an example of a captured image 600, which is obtainedby applying the present system to the captured image 300 and in whichpatterns are superimposed. In FIG. 7, solid-line ellipses 601 to 604indicate a pattern representing a collaborative worker, and adouble-line ellipse 605 indicates a pattern representing an isolatedworker.

In the present exemplary embodiment, a solid-line ellipse is used as apattern representing a collaborative worker. Moreover, a double-lineellipse is used as a pattern representing an isolated worker. However,this is not limiting, and a collaborative worker and an isolated workercan be distinguished by colors of lines.

Furthermore, each collaborative worker and each isolated worker can besurrounded with a solid-line ellipse, and ellipses serving as a pair ofcollaborative workers can be connected to each other by a line, or agroup of persons serving as a pair of collaborative workers can besurrounded with a large circle. Additionally, workers can be connectedto each other by a line without being surrounded with an ellipse.

Furthermore, groups of collaborative workers do not need to be assignedthe same pattern, and, likewise, isolated workers do not need to beassigned the same pattern. Different patterns can be applied torespective groups of collaborative workers or respective isolatedworkers. For example, in the present exemplary embodiment, the controlunit 121 can set the ellipses 601 and 602, which represent acollaborative worker, as a red ellipse, set the ellipses 603 and 604,which also represent a collaborative worker, as a blue ellipse, and setthe ellipse 605, which represents an isolated worker, as a greenellipse.

Moreover, while, in the present exemplary embodiment, ellipses are usedas patterns representing a collaborative worker and an isolated worker,this is not limiting. For example, a collaborative worker can berepresented by a square, and an isolated worker can be represented by atriangle.

Furthermore, numerals or character strings for discriminating between anisolated worker and a collaborative worker can be appended. For example,the control unit 121 can superimpose a character string “isolated worker1” on the vicinity of the position of a person determined to be anisolated worker on the captured image, and can surround a plurality ofpersons determined to be collaborative workers with a circle andsuperimpose a character string “collaborative workers 1” on the vicinityof the circle. In a case where a plurality of isolated workers or aplurality of groups of collaborative workers is present, the controlunit 121 can superimpose character strings the numerals of ends of whichare varied.

Additionally, if attention needs to be paid to only an isolated worker,the control unit 121 can be configured not to superimpose a patternrepresenting a collaborative worker, or can be configured to hide such apattern even when superimposing the pattern.

FIG. 8 illustrates an example of a horizontal plane image 700, which isobtained by applying the present system to the horizontal plane image500 and in which patterns representing a collaborative worker and anisolated worker are superimposed. In FIG. 8, solid-line circles 701 to704 indicate a pattern representing a collaborative worker, and adouble-line circle 705 indicates a pattern representing an isolatedworker. Even in a horizontal plane image, like a captured image, apattern representing an isolated worker and a pattern representing acollaborative worker does not need to be distinguished by the types oflines. In other words, a pattern representing an isolated worker and apattern representing a collaborative worker can be distinguished bycolors, types of graphics, numerals, or character strings. Moreover,indexes representing workers (in FIG. 8, circles 701 to 704) can beconnected to each other by a line.

Next, in step S210, the control unit 121 controls the display controlunit 125 to cause the display device 140 to perform display processingfor displaying a processing result obtained by superimposing patterns onthe image. In the present exemplary embodiment, the captured image 600and the horizontal plane image 700, in which a pattern representing anisolated worker and a pattern representing a collaborative worker aresuperimposed, are displayed on the display device 140.

However, this is not limiting, but the control unit 121 can cause thedisplay device 140 to display only the captured image 600, in which thepatterns are superimposed, or can cause the display device 140 todisplay only the horizontal plane image 700, in which the patterns aresuperimposed.

Furthermore, the control unit 121 can make an association between thecaptured image 600 and the horizontal plane image 700. For example, thecontrol unit 121 can assign the same number to a person in the capturedimage 600 and a position obtained by coordinate conversion of the personon the horizontal plane image 700, or can append the same characterstring thereto. Additionally, the control unit 121 can perform controlto allow the user to specify the position of a person in the horizontalplane image 700 using the input device 130 so as to highlight a patternin the captured image 600 corresponding to the specified index. Themethod for highlighting a pattern includes, for example, thickening aline of the pattern and changing the color of the line.

Moreover, while, in the present exemplary embodiment, the control unit121 performs control to generate images in which patterns aresuperimposed on the captured image 600 and the horizontal plane image700 and to cause the display device 140 to display the generated images,this is not limiting. The control unit 121 can control the displaycontrol unit 125 to cause the display unit 140 to superimpose images ofa pattern representing an isolated worker and a pattern representing acollaborative worker on the captured image 600 and the horizontal planeimage 700 and to cause the display device 140 to display them.

Furthermore, while, in the present exemplary embodiment, an example hasbeen described in which a single client apparatus detects, for example,persons and displays the persons, a plurality of client apparatuses canperform such processing. For example, processing can be performed by afirst client apparatus, which receives a video, detects, for example, anisolated worker, and sends a notification thereof, and a second clientapparatus, which receives the notification to output an alarm or toperform superimposition display. An example of the first clientapparatus is a video content analysis (VCA) apparatus. An example of thesecond client apparatus is a video management system (VMS) apparatus.The hardware configuration of the first client apparatus and the secondclient apparatus is similar to that of the client apparatus 120illustrated in FIG. 1. The functions of the respective clientapparatuses are implemented by the CPUs of the respective clientapparatuses performing processing based on programs stored in memoriesof the respective client apparatuses.

As described above, according to processing in the present exemplaryembodiment, the client apparatus 120 is able to detect persons from animage and to associate the detected persons with each other based on atleast one of positions of the persons, directions of faces of thepersons, and distances between the persons. Then, the client apparatus120 is able to cause a display device to perform display processing forenabling discriminating between a person who is associated with anotherperson and a person who is not associated with any other person. As aresult, information indicating whether a person shown in an image isisolated from or associated with another person can be appropriatelypresented.

In the first exemplary embodiment, for example, distances betweencoordinates on the horizontal plane image are used to determine whethera person is an isolated worker or a collaborative worker. Moreover,patterns representing an isolated worker or a collaborative worker aredisplayed on the captured image or the horizontal plane image. However,this is not limiting. In a second exemplary embodiment, a case isdescribed in which detection areas are used to determine whether thetarget person is an isolated worker or a collaborative worker and todisplay patterns representing an isolated worker or a collaborativeworker.

In the second exemplary embodiment, elements or steps similar to thoseof the first exemplary embodiment are omitted from the description, andelements or steps different from those of the first exemplary embodimentare mainly described.

In the present exemplary embodiment, in step S205, the control unit 121determines whether a person of interest is an isolated worker usingdetection areas. For example, the control unit 121 sets a plurality ofdetection areas on the captured image and the horizontal plane imagebased on instructions input by the user via, for example, the inputdevice 130. At this time, in step S205, in a case where another personis present within a detection area in which a person targeted forprocessing is present, the control unit 121 determines that the persontargeted for processing is a collaborative worker. On the other hand, ina case where no other person is present within the detection area, thecontrol unit 121 determines that the person targeted for processing isan isolated worker.

Then, the control unit 121 superimposes a pattern representing acollaborative worker on a detection area in which a collaborative workeris present, and superimposes a pattern representing an isolated workeron a detection area in which an isolated worker is present.

The above-described processing is described in detail with reference toFIGS. 9 and 10. FIG. 9 illustrates an example of a captured image 800 inwhich detection areas are set. Moreover, FIG. 10 illustrates an exampleof a horizontal plane image 900 in which detection areas are set.Detection areas 801 to 806 and 901 to 906 represent areas in which agroup of collaborative workers may be present. In the examplesillustrated in FIGS. 9 and 10, a plurality of persons is detected ineach of the detection areas 801, 805, 901 and 905. At this time, if itis determined whether the target person is an isolated worker in thepresent exemplary embodiment, it is determined that the persons presentin the detection areas 801, 805, 901 and 905 are collaborative workers.On the other hand, only one person is detected in each of the detectionareas 803 and 903. At this time, if the control unit 121 in the presentexemplary embodiment determines whether the target person is an isolatedworker, the control unit 121 determines that the person present in thedetection areas 803 and 903 is an isolated worker. Furthermore, as aresult of the above determination, the control unit 121 controls thedisplay device 140 to perform display processing in such a way as toexpress the boundaries of the detection areas 801, 805, 901 and 905 bysolid lines and to express boundaries of the detection areas 803 and 903by double lines. This enables expressing whether a person present ineach detection area is an isolated worker or a collaborative worker.

While, in the present exemplary embodiment, changing the boundary of adetection area to a solid line or a double line is used to expresswhether a person present in each detection area is an isolated worker ora collaborative worker, this is not limiting. A pattern representing anisolated worker and a pattern representing a collaborative worker do notneed to be distinguished by the types of lines. Those can bedistinguished by colors, or can be distinguished by different graphics.

In the present exemplary embodiment, a description has been made usingan example in which the control unit 121 sets detection areas. Whendetecting an isolated worker, the control unit 121 can be configured tosuperimpose a detection area in which the isolated worker has beendetected on a video and to cause the display device 140 to display thevideo with the superimposed detection area. Moreover, the control unit121 can be configured to first superimpose all detection areas on avideo to display the video with the superimposed detection areas, and,when detecting an isolated worker, to display a detection area in whichthe isolated worker has been detected in such an explicit or highlightedmanner as to change the color of the detection area to a color differentfrom those of the other detection areas or to perform blinking display.This enables the work observer to understand, with good visibility,whether a person shown in a video is isolated from or associated withanother person.

Furthermore, a clipped image, which is obtained by clipping an areacorresponding to a person shown in the captured image, can be displayed.Then, a clipped image corresponding to an isolated worker and a clippedimage corresponding to a collaborative worker can be displayed in adiscriminable manner. For example, each clipped image can be surroundedwith a frame and the colors of frames can be set in such a way as tovary between the clipped image corresponding to an isolated worker andthe clipped image corresponding to a collaborative worker. If only anarea corresponding to a person is clipped, observation becomes easy.Moreover, in a case where a plurality of imaging apparatuses is used,the image of a single person may be captured by the plurality of imagingapparatuses. In that case, such a configuration that only a capturedimage obtained by one of the imaging apparatuses is displayed can beemployed, so that observation also becomes easy.

Next, a third exemplary embodiment is described with reference to FIG.11. In the present exemplary embodiment, elements or steps similar tothose of the other exemplary embodiments are omitted from thedescription as appropriate, and elements or steps different from thoseof the other exemplary embodiments are mainly described.

In the flowchart of FIG. 11, in step S1303, the control unit 121 updatesa list of detected persons. In the case of the first cycle of repetitiveprocessing, the control unit 121 lists all of the detected persons. Atthis time, the control unit 121 stores, in each node in the list, thecoordinates of a representative point on the captured image at which aperson has been detected and the coordinates of a representative pointon the horizontal plane image at which the person has been detected. Inthe case of the second and subsequent cycles of repetitive processing,the control unit 121 performs the following processing on all of thedetected persons.

(Processing 1). The control unit 121 calculates a distance between thecoordinates on the captured image of a person stored in each node in thelist and the coordinates on the captured image of itself, and, if thecalculated distance is equal to or less than a threshold value, replacesa target node with data of itself. At this time, the control unit 121stores a history of coordinates which have been stored in the nodesuntil now.

(Processing 2). If there is no node satisfying the condition in(Processing 1), the control unit 121 adds data of itself to the last ofthe list. Moreover, after completing the above processing on all of thedetected persons, the control unit 121 performs the followingprocessing.

(Processing 3). The control unit 121 deletes a node or nodes that havenot been updated.

Next, in steps S1304 to S1306, the control unit 121 determines whetheran isolated worker is present in the captured image. The flow ofprocessing is as follows.

(Processing 1). In step S1304, the control unit 121 sets the head of thelist as a processing target.

(Processing 2). In step S1305, the control unit 121 determines whether aperson serving as a processing target is an isolated worker. In thepresent exemplary embodiment, the control unit 121 checks the followingitems with respect to the person serving as a processing target so as todetermine whether that person is an isolated worker.

(Item 1). The control unit 121 sets the coordinates on the horizontalplane image of the person serving as a processing target as the center,and determines whether a detected person is present within the radius Rfrom the center.

The radius R can be set as appropriate according to a situation to whichthe present exemplary embodiment is applied. For example, in the case ofobserving an indoor work, the radius R can be set to a distancecorresponding to 2 m. However, at this time, the control unit 121 refersto a history of coordinates stored in the node of the person serving asa processing target, and sets the value of the radius R, which is athreshold value, larger when determining that the person serving as aprocessing target is in motion than when determining that the personserving as a processing target is not in motion. Furthermore, at thistime, if collaborative workers are working side by side along the x-axisdirection on the horizontal plane image, the control unit 121 can use anellipse obtained by shortening a circle with the radius R in the y-axisdirection. Similarly, if collaborative workers are working side by sidealong the y-axis direction on the horizontal plane image, the controlunit 121 can use an ellipse obtained by shortening a circle with theradius R in the x-axis direction. The control unit 121 sets a personmeeting the condition of (Item 1) as a candidate for a collaborativeworker, and checks the following (Item 2) and (Item 3).

(Item 2). In a case where a person satisfying (Item 1) is present, thecontrol unit 121 determines whether a half line indicating the directionof the face of that person intersects with a half line indicating thedirection of the face of the person serving as a processing target.

However, at this time, if the direction of the face of that person hasnot been detected, the control unit 121 does not take (Item 2) indetermining whether the person serving as a processing target is anisolated worker or a collaborative worker.

(Item 3). In a case where a person satisfying (Item 1) is present, thecontrol unit 121 determines whether that person is present outside anon-detection area.

In the present exemplary embodiment, the control unit 121 sets an areaspecified by the user operating, for example, the input device 130 as anon-detection area, in which any person cannot be present. This enablespreventing, for example, a non-worker who is moving on the opposite sideof a glass-made wall or a person shown on, for example, a poster placedon a desk from being erroneously detected as a collaborative worker.Setting of the non-detection area is not limited to inputting by theuser via, for example, the input device 130. The control unit 121 canautomatically set an area surrounded with a marker laid on the capturedimage as a non-detection area. The method for detection using a markerincludes, for example, performing pattern matching betweentwo-dimensional markers set at the four corners of a non-detection areaand information on the above two-dimensional markers previouslyregistered with the client apparatus 120 by performing raster scanningon the captured image.

In a case where all of the above-mentioned (Item 1) to (Item 3) aresatisfied, the control unit 121 determines that the person serving as aprocessing target is a collaborative worker (NO in step S1305), and thenproceeds to (Processing 3-1) (S1306). In other words, the control unit121 associates workers satisfying all of the above-mentioned (Item 1) to(Item 3) with each other and determines that the workers arecollaborative workers. In this way, the control unit 121 functions as anassociation unit. Moreover, in a case where at least one of theabove-mentioned (Item 1) to (Item 3) is not satisfied, the control unit121 determines that the person serving as a processing target is anisolated worker (YES in step S1305), and then proceeds to (Processing3-2) (step S1309).

Here, processing for determining whether the person serving as aprocessing target is an isolated worker or a collaborative worker is notlimited to the above-described processing. For example, the control unit121 can be configured to associate persons with each other and determinethat the associated person is a collaborative worker based on at leastone of a position of one or more persons detected from the capturedimage, directions of faces of the persons included in the image, anddistances between the persons included in the image. For example, thecontrol unit 121 can be configured to associate persons present within apredetermined range from a predetermined object, such as a work table,in the captured image with each other, or can be configured to estimatedirections (or eye lines) of faces of persons and to associate personsthe directions of faces of whom intersect with each other. Moreover, forexample, the control unit 121 can be configured to associate persons thedistance between whom is within a predetermined distance with eachother. Then, the control unit 121 can be configured to determine thatpersons each of whom is associated with another person are collaborativeworkers and a person who is not associated with any other person is anisolated worker. Additionally, the control unit 121 can be furtherconfigured to associate persons with each other based on colors orshapes of uniforms, bibs (vest-like clothes), or caps of persons. Forexample, the control unit 121 can be configured to associate persons theshapes of uniforms of whom are the same among persons the distancebetween whom is within a predetermined distance with each other, and notto associate persons the shapes of uniforms of whom are different evenamong persons the distance between whom is within a predetermineddistance with each other.

The determinations in the above-mentioned (Item 1) to (Item 3) aredescribed with reference to FIG. 12. In FIG. 12, coordinates 1401 to1405 represent results of converting the centers of the ellipses 401 to405, which are person detection positions, into coordinates on thehorizontal plane image. Moreover, regions 1406 and 1407 representnon-detection areas specified by the user. Additionally, an ellipse 1400represents an ellipse defined by points x and y satisfying the followingformula (4):

(x−qx)² /a ²+(y−qy)² /b ² =R ¹  (4)

In formula (4), qx and qy denote the x coordinate and y coordinate,respectively, on the horizontal plane image of a person serving as aprocessing target. In FIG. 12, qx and qy represent the x coordinate andy coordinate, respectively, of the coordinates of a circle 1402. Informula (4), a and b denote transformation parameters for an ellipse.For example, when a=1 and b=1, the ellipse becomes a circle with aradius of R and a center of (qx, qy). Moreover, when a=1 and b=½, theellipse becomes an ellipse with a length of R in the x-axis directionand a length of R/2 in the y-axis direction. In the present exemplaryembodiment, since collaborative workers are working side by side alongthe x-axis direction on the horizontal plane image, the values of thetransformation parameters are set to a relationship of “a>b”.

For example, suppose that the control unit 121 sets a person in theellipse 402, which is a person detection position, as a processingtarget during scanning of the list. At this time, the control unit 121performs determinations in the above-mentioned (Item 1) to (Item 3) withrespect to the coordinates of circle 1402 on the horizontal plane imagecorresponding to the ellipse 402 of FIG. 5, which is a person detectionposition.

First, in (Item 1), the control unit 121 determines whether a detectedperson is present inside the ellipse 1400. In the situation illustratedin FIG. 12, since the coordinates 1401 on the horizontal plane imagecorresponding to the ellipse 401 for person detection are included inthe ellipse 1400, the control unit 121 determines that the requiredcondition is satisfied.

Next, in (Item 2), the control unit 121 determines whether a half line(arrow) 406 representing the direction of the face of the ellipse 402,which is a person detection position, and an arrow 407 representing thedirection of the face of the ellipse 401, which is a person detectionposition of a person causing satisfaction of the condition of (Item 1),intersect. In the situation described in the present exemplaryembodiment, since the arrows 406 and 407 intersect, the control unit 121determines that the required condition is satisfied.

Next, in (Item 3), the control unit 121 determines whether thecoordinates 1401 on the horizontal plane image corresponding to theellipse 401, which is a person detection position of a person causingsatisfaction of the condition of (Item 1), are not included in any ofthe areas 1406 and 1407, which are non-detection areas. In the situationdescribed in the present exemplary embodiment, since the coordinates1401 are not included in any of the areas 1406 and 1407, which arenon-detection areas, the control unit 121 determines that the requiredcondition is satisfied. Thus, since all of the requirements in (Item 1)to (Item 3) are satisfied, the control unit 121 determines that theperson in the ellipse 402, which is a person detection position, is acollaborative worker.

Furthermore, in the present exemplary embodiment, the control unit 121sets a person satisfying the condition of (Item 1) as a candidate for acollaborative worker. However, this is not limiting, but the controlunit 121 can perform determination based on specifying of areas. Forexample, the control unit 121 sets a plurality of areas in which a pairof workers may be present on the captured image or the horizontal planeimage based on instructions input by the user. At this time, the controlunit 121 can set the condition of (Item 1) to whether a person ispresent in the same detection area as that of a person of interest.

(Processing 3-1). In step S1306, the control unit 121 determines whetherthe last of the list is set as a processing target.

If the processing target is not the last of the list (NO in step S1306),then in step S1307, the control unit 121 sets a next person as aprocessing target, and then returns to (Processing 2). If the processingtarget is the last of the list (YES in step S1306), the control unit 121ends scanning on the list, and then in step S1308, sets the cumulativenumber of frames in which an isolated worker is present to “0”.

(Processing 3-2). In step S1309, the control unit 121 increments thecumulative number of frames in which an isolated worker is present, inother words, increases the value of a variable by one.

Next, in steps S1310 and S1311, the control unit 121 determines whetherto notify the user that an isolated worker is present, and, if thatcondition is satisfied, sends a notification to that effect. In thepresent exemplary embodiment, in a case where the cumulative number offrames (cumulative period) in which an isolated worker is presentexceeds a threshold value, the control unit 121 sends a warning to theuser. Using this determination condition enables preventing, forexample, erroneously notifying the user that the person serving as aprocessing target is an isolated worker due to a momentary failure ofperson detection despite of being a collaborative worker. Moreover, thecontrol unit 121 can be configured to send a warning to the user when aperiod in which an isolated worker is present exceeds a threshold valueinstead of making a determination based on the cumulative number offrames. The control unit 121 can send a warning to the user in a casewhere a period (the cumulative number of frames or time) in which anisolated worker is present exceeds a threshold value. Furthermore, thecontrol unit 121 can send a warning to the user in a case where a period(continuous period) for which a state in which an isolated worker ispresent continues exceeds a predetermined threshold value. Here, thethreshold value is stored in, for example, the memory 21. The controlunit 121 is able to change the threshold value according to a settingoperation performed by the user via, for example, the input device 130.Additionally, the threshold value can be set to “0”. Thus, in a casewhere an isolated worker is present, the user can be immediatelynotified.

Furthermore, in the present exemplary embodiment, the method fornotifying the user that an isolated worker is present is describedusing, as an example, a method of displaying, on the display device 140,a user interface (UI) indicating that an isolated worker is present. Forexample, the control unit 121 causes the display device 140 via thedisplay control unit 125 to perform display processing, such assuperimposing a double-line circle or superimposing a red circle at theposition of an isolated worker on the captured image 300 or thehorizontal plane image 500. However, this is not limiting, but thecontrol unit 121 can, for example, send an e-mail to the user or canissue an alert from a speaker. Additionally, the control unit 121 canissue a notification by turning on a light emission unit, such as alight-emitting diode (LED).

Next, in step S1312, the control unit 121 determines whether to repeatprocessing from acquisition of a captured image (step S200) todetermination of whether to notify the user of the presence or absenceof an isolated worker (steps S1310 and S1311). In the present exemplaryembodiment, the control unit 121 determines to end repetitive processingin a case where the acquisition of a captured image from the imagingapparatus 110 has become impossible or in a case where an instructionfor stopping the repetitive processing has been issued by the user.

As described above, if, in a moving image, there is a period of a movingimage in which an isolated worker, who is not associated with any otherperson, is present, notification processing is performed according tosuch a period. This enables informing an observer that an isolatedworker is present.

In the present exemplary embodiment, a description has been made mainlyusing an example in which the control unit 121 sets detection areas.When detecting an isolated worker, the control unit 121 can beconfigured to superimpose a detection area in which the isolated workerhas been detected on a video and to cause the display device 140 todisplay the video with the superimposed detection area. Moreover, thecontrol unit 121 can be configured to first superimpose all detectionareas on a video to display the video with the superimposed detectionareas, and, when detecting an isolated worker, to display a detectionarea in which the isolated worker has been detected in such an explicitor highlighted manner as to change the color of the detection area to acolor different from other detection areas or to perform blinkingdisplay. This enables the work observer to understand, with goodvisibility, whether a person shown in a video is isolated from orassociated with another person.

Furthermore, while, in the present exemplary embodiment, an example hasbeen described in which a single client apparatus detects, for example,persons and displays the persons, a plurality of client apparatuses canperform such processing. For example, processing can be performed by afirst client apparatus, which receives a video, detects, for example, anisolated worker, and sends a notification thereof, and a second clientapparatus, which receives the notification to output an alarm or toperform superimposition display. An example of the first clientapparatus is a video content analysis (VCA) apparatus. An example of thesecond client apparatus is a video management system (VMS) apparatus.The hardware configuration of the first client apparatus and the secondclient apparatus is similar to that of the client apparatus 120illustrated in FIG. 1. The functions of the respective clientapparatuses are implemented by the CPUs of the respective clientapparatuses performing processing based on programs stored in memoriesof the respective client apparatuses.

When receiving a warning indicating that an isolated worker is presentat a work site, the work observer can find a worker breaking a rule andthus can call attention to the worker.

Next, a fourth exemplary embodiment is described with reference to FIG.13. In the present exemplary embodiment, elements or steps similar tothose of the other exemplary embodiments are omitted from thedescription as appropriate, and elements or steps different from thoseof the other exemplary embodiments are mainly described.

In step S1503, the control unit 121 updates a list of detected persons.In the case of the first cycle of repetitive processing, the controlunit 121 lists all of the detected persons. At this time, the controlunit 121 stores, in each node in the list, the coordinates of arepresentative point on the captured image at which a person has beendetected and the coordinates of a representative point on the horizontalplane image at which the person has been detected. In the case of thesecond and subsequent cycles of repetitive processing, the control unit121 performs the following processing on all of the detected persons.

(Processing 1). The control unit 121 calculates a distance between thecoordinates on the captured image of a person stored in each node in thelist and the coordinates on the captured image of itself, and, if thecalculated distance is equal to or less than a threshold value, replacesa target node with data of itself. At this time, the control unit 121stores a history of coordinates which have been stored in the nodesuntil now.

(Processing 2). If there is no node satisfying the condition in(Processing 1), the control unit 121 adds data of itself to the last ofthe list.

Moreover, after completing the above processing on all of the detectedpersons, the control unit 121 performs the following processing.

(Processing 3). The control unit 121 deletes a node or nodes that havenot been updated.

Next, in steps S1504 to S1506, the control unit 121 determines whetheran isolated worker is present in the captured image. In the presentexemplary embodiment, as illustrated in FIG. 14A, the control unit 121sets work area frames (901 to 906) in a captured image display based on,for example, a user operation. The flow of processing is as follows.

(Processing 4). In step S1504, the control unit 121 checks whether thecoordinates of a person set as a processing target in order from thehead of the list are present in a work area frame, and, if thecoordinates of the person are present in the work area frame, stores, inthe list, a flag indicating that the coordinates of the person arepresent in the work area frame. At this time, if there is a plurality ofwork areas, the control unit 121 stores, in the list, identification(ID) numbers which are assigned for the respective work area frames in anon-overlapping way.

(Processing 5). In step S1505, the control unit 121 searches the listand checks how many workers belonging to the area of each work areaframe are present. In a case where only one worker is present in a workarea frame (YES in step S1506), the control unit 121 determines that anisolated work is being performed in the work area frame, and, in theother cases (NO in step S1506), the control unit 121 determines that noisolated work is being performed in the work area frame. In other words,in a case where only one worker is present in a work area frame, thecontrol unit 121 determines that the worker present in the correspondingarea is an isolated worker, who is not associated with any other person.In a case where two or more workers are present in a work area frame,the control unit 121 associates the workers present in the correspondingarea with each other and determines that those workers are collaborativeworkers. In this way, the control unit 121 functions as an associationunit.

While, in the present exemplary embodiment, the control unit 121 firstdetermines whether a worker is present in a work area frame based on alist of detected persons, the control unit 121 can check whether eachperson is present in a work area frame based on each work area frame andmake such a determination based on only the number of persons present ineach work area.

Furthermore, while a description has been made using an example in whichwork area frames are set in a captured image display, the control unit121 can set work area frames in a horizontal plane image based on a useroperation, as illustrated in FIG. 14B.

Here, processing for determining whether the person of interest is anisolated worker or a collaborative worker is not limited to theabove-described processing. For example, the control unit 121 can beconfigured to associate persons with each other and determine that theassociated person is a collaborative worker based on at least one ofpositions of persons detected from the captured image, directions offaces of the persons, and distances between the persons. For example,the control unit 121 can be configured to associate persons presentwithin a predetermined range from a predetermined object, such as a worktable, in the captured image with each other, or can be configured toestimate directions (or eye lines) of faces of persons and to associatepersons the directions of faces of whom intersect with each other.Moreover, for example, the control unit 121 can be configured toassociate persons the distance between whom is within a predetermineddistance with each other. Then, the control unit 121 can be configuredto determine that persons each of whom is associated with another personare collaborative workers and a person who is not associated with anyother person is an isolated worker.

Additionally, the control unit 121 can be further configured toassociate persons with each other based on colors or shapes of uniforms,bibs (vest-like clothes), or caps of persons. For example, the controlunit 121 can be configured to associate persons the shapes of uniformsof whom are the same among persons the distance between whom is within apredetermined distance with each other, and not to associate persons theshapes of uniforms of whom are different even among persons the distancebetween whom is within a predetermined distance with each other. In thisway, the control unit 121 can be configured to perform association ofpersons based on feature amounts of things worn by the persons.

Furthermore, even in a case where a plurality of workers is present in awork area frame, the control unit 121 can be configured to determinethat each worker is an isolated worker depending on directions of facesof the workers in the work area frame or distances between the workers.

Next, in steps S1507 to S1509, the control unit 121 determines whetheran isolated work is being performed, and, if that condition issatisfied, records a log indicating the isolated work and sends anotification of the log. The flow of processing is as follows.

(Processing 6). In a case where a work area frame in which an isolatedwork is being performed is present in (Processing 5), then in stepS1507, the control unit 121 increments the cumulative number of isolatedwork frames, in other words, increases the value of a variable by one,and extracts a person belonging to the corresponding work area from thelist. In a case where no isolated work is being performed, then in stepS1508, the control unit 121 sets the cumulative number of isolated workframes to “0”.

(Processing 7). If the cumulative number of isolated work frames exceedsa threshold value (YES in step S1509), the control unit 121 determinesthat an isolated work is being performed. Using this determinationcondition enables preventing, for example, erroneously notifying theuser that the target worker is an isolated worker due to a momentaryfailure of person detection despite of being a collaborative worker.

If it is determined that an isolated work is being performed, then instep S1510, the control unit 121 displays, on the screen of the displaydevice 140, such a notification as to indicate the clock time ofoccurrence of the isolated work, the duration thereof, the imagingapparatus 110 capturing an image of the isolated work, and, for example,the location at which the isolated work is being performed in thecaptured image. The screen illustrated in FIG. 15A is an example of ascreen displaying a list of moving images captured in a time period forwhich an isolated work is detected. Text information written as“Isolated Work Detection List” and information on the clock time,duration, and location illustrated in FIG. 15A are an example ofinformation distinguishably indicating a moving image in which a personwho is not associated with any other person is contained. Moreover,depending on the purpose of use, moving images captured in time periodsin which a person who is not associated with any other person is notcontained can be displayed as a list. In that case, for example, textinformation written as “Collaborative Work Detection List” is displayed.

Next, in step S1511, the control unit 121 determines whether to repeatprocessing from acquisition of a captured image (step S200) todetermination of whether to notify the user of the presence or absenceof an isolated worker (steps S1509 and S1510). The control unit 121determines to end repetitive processing in a case where the acquisitionof a captured image has become impossible or in a case where aninstruction for stopping the repetitive processing has been issued bythe user.

While, in the display provided in step S1510, the control unit 121 isconfigured to display thumbnail images together with information (list)about isolated works as illustrated in FIG. 15A, the control unit 121can display the information only in a list form. Moreover, the controlunit 121 can display timelines together with display of a recorded videoas illustrated in FIG. 15B, and can display an indication representingthe time period of an isolated work on each timeline. A portion (a solidblack portion or a shaded portion) of a timeline represented as a timeperiod of an isolated work illustrated in FIG. 15B is an example ofinformation distinguishably indicating a moving image in which a personwho is not associated with any other person is contained. Furthermore,for example, the solid black portion represents a time period for whichonly one isolated worker is present, and the shaded portion represents atime period for which a plurality of isolated workers is present.Additionally, “A-1” is information indicating, for example, an ID foridentifying an imaging apparatus. In the example illustrated in FIG.15B, timelines “A-1” and “B-1” in two imaging apparatuses areillustrated.

On the other hand, a portion (a white portion) of a timeline representedas not a time period of an isolated work illustrated in FIG. 15B is anexample of information distinguishably indicating a moving image inwhich a person who is not associated with any other person is notcontained. Furthermore, the control unit 121 can display time periods ofan isolated work detected by a plurality of cameras on a singletimeline. As mentioned above, at least one of informationdistinguishably indicating that a person who is not associated with anyother person is contained and information distinguishably indicatingthat a person who is not associated with any other person is notcontained can be displayed on at least a timeline. Thus, a time periodfor which a person who is not associated with any other person iscontained and a time period for which a person who is not associatedwith any other person is not contained, in a moving image, can bedistinguishably indicated.

Furthermore, for example, the user can specify display of each isolatedwork with a mouse or finger, as illustrated in FIG. 16A. Then, thecontrol unit 121 can cause the display device 140 via the displaycontrol unit 125 to play back a video recorded by the correspondingimaging apparatus to display a captured image of the played-back video,as illustrated in FIG. 16B, thus enabling checking the captured image.This can increase the user's convenience. At this time, while thecontrol unit 121 can play back only a video recorded at the time ofoccurrence of an isolated work, the control unit 121 can desirably playback a video starting with a video recorded a predetermined time (forexample, several tens of seconds) before the time of occurrence of anisolated work or can desirably play back a video recorded for a timelonger than a duration of an isolated work, thus enabling easilychecking the situations before and after the isolated work.

Moreover, the control unit 121 can join videos recorded in time periodsdetected as isolated works, and then can play back the joined videos asa digest video. In other words, the control unit 121 can be configuredto play back moving images in the list illustrated in FIG. 16A in acontinuous manner. At that time, a moving image in a time period of anisolated work in the order of several seconds can be omitted from thedigest image.

While, to prevent any false detection in notification of an isolatedwork, a warning is issued after the isolated work continues for apredetermined time, a duration of the isolated work is a time periodstarting with the first detected frame. Accordingly, the control unit121 can set the start of a time interval displayed as an isolated workin a timeline or the start of recording or playback of a video to notthe time when a warning is issued but the time when an isolated workbegins to be detected.

Furthermore, the control unit 121 not only displays a timeline as anisolated work time period detected in a case where a warning is issuedafter the cumulative number of frames in which an isolated worker ispresent exceeds a threshold value, but also can perform the followingprocessing. For example, even in a case where it is determined that anisolated work is being performed for such a short period as not to causea warning, the control unit 121 can provide a display with a color,shape, or design different from that for an isolated work time periodcausing a warning, thus enabling checking isolated works for both thetime periods. In this case, since an isolated work that is onlymomentarily detected may be detected merely when, in actuality, anyperson of collaborative workers is temporarily hidden by a shieldingobject, two threshold values can be provided for determination of anisolated work. At this time, the control unit 121 issues a warning basedon a threshold value that is the larger of the two threshold values. Forexample, with respect to information distinguishably indicating that aperson who is not associated with any other person is contained, thecontrol unit 121 uses a first threshold value, for example, 10 seconds.Then, to determine whether to issue a warning, the control unit 121 usesa second threshold value larger than the first threshold value, forexample, 20 seconds.

Moreover, the control unit 121 can be configured to store, as a commaseparated value (CSV) file, a list of imaging apparatuses, areas, andtime information corresponding to detected isolated works, or can beconfigured to paste, as a report, a list with thumbnail images or screenshots of a recorded video to be ready for printing.

Additionally, in a case where, besides a captured image used fordetection of an isolated work, there is provided another imagingapparatus which captures an image of a similar area at another angle orcaptures an image of a surrounding area, the control unit 121 can beconfigured to perform the following processing. For example, the controlunit 121 can display a video captured and recorded by an imagingapparatus on which the setting of association has been performed inadvance.

In addition, in the case of a system which detects an isolated workusing live videos and, when determining that the isolated work is beingperformed, issues a warning in real time, as illustrated in FIG. 17A,the work observer needs to not only check the current situation but alsocheck the past situation. Therefore, the control unit 121 can performcontrol to display both a live video and a recorded video. Referring toFIG. 17A, in a video showing a room A, changing an index located at theupper right corner corresponds to displaying information distinguishablyindicating that a person who is not associated with any other person iscontained.

The control unit 121 can be configured to erase, from a series ofdisplayed images, for example, an image determined and selected by thework observer to be free from any false notification, as illustrated inFIG. 17B.

In the present exemplary embodiment, a description has been made usingan example in which the control unit 121 sets work area frames. Thecontrol unit 121 can be configured to, when detecting an isolated work,cause the display device 140 to display a work area frame in which theisolated work is detected in superimposition on a video. Furthermore,the control unit 121 can previously display all work area frames insuperimposition on a video and, when an isolated work is performed,display a work area frame in which the isolated work is performed insuch an explicit or highlighted manner as to change the color of thework area frame to a color different from those of the other work areaframes or to perform blinking display. This enables the work observer tounderstand, with good visibility, whether a person shown in a video isisolated from or associated with another person.

Furthermore, while, in the present exemplary embodiment, an example hasbeen described in which a single client apparatus detects, for example,persons and displays the persons, a plurality of client apparatuses canperform such processing. For example, processing can be performed by afirst client apparatus, which receives a video, detects, for example, anisolated worker, and sends a notification thereof, and a second clientapparatus, which receives the notification to output an alarm or toperform list display or timeline display. An example of the first clientapparatus is a video content analysis (VCA) apparatus. An example of thesecond client apparatus is a video management system (VMS) apparatus.The hardware configuration of the first client apparatus and the secondclient apparatus is similar to that of the client apparatus 120illustrated in FIG. 1. The functions of the respective clientapparatuses are implemented by the CPUs of the respective clientapparatuses performing processing based on programs stored in memoriesof the respective client apparatuses.

FIG. 18A illustrates an example of a case where the system according tothe present exemplary embodiment and an entrance and exit managementsystem 160 cooperate with each other. The entrance and exit managementsystem 160 records the clock times at which workers enter and exit aworkroom. Accordingly, as illustrated in FIG. 18B, the control unit 121acquires information on the clock times at which each worker enters andexits the workroom from the entrance and exit management system 160, andextracts workers who are present in the workroom from the acquiredinformation. Then, the control unit 121 can be configured to record alist of names of the extracted workers as a part of informationindicating a list of occurrences of an isolated work or to display sucha list on a screen. However, this is not limiting, but the control unit121 can send an e-mail to the user. This recording or displaying can beimplemented by the system according to the present exemplary embodiment,or can be incorporated into, for example, the entrance and exitmanagement system 160. The list of names of the extracted workers is anexample of entrance and exit information.

A fifth exemplary embodiment is the same as the other exemplaryembodiments in a part of the configuration thereof. In the presentexemplary embodiment, a description is made using an example in which adetermination of whether an isolated worker is present is made not basedon detection using a work area frame but based on distances. The flow ofprocessing in the present exemplary embodiment is described withreference to FIG. 19. Moreover, elements or steps similar to those ofthe other exemplary embodiments are omitted from the description, andelements or steps different from those of the other exemplaryembodiments are mainly described.

Processing performed from the time when the control unit 121 acquires acaptured image from the imaging apparatus 110 (step S1101) to the timewhen the control unit 121 updates a list of detected persons (stepS1103) is similar to the processing described in the first exemplaryembodiment.

Next, in steps S1104 to S1106, the control unit 121 determines whetheran isolated worker is present in the captured image. The flow ofprocessing is as follows.

(Processing 1). In step S1104, the control unit 121 sets the head of thelist as a processing target.

(Processing 2). In step S1105, the control unit 121 determines whether aperson serving as a processing target is an isolated worker. In thepresent exemplary embodiment, the control unit 121 checks the followingitems with respect to the person serving as a processing target so as todetermine whether that person is an isolated worker.

(Item 1). The control unit 121 sets the coordinates on the horizontalplane image of the person serving as a processing target as the center,and determines whether a detected person is present within the radius Rfrom the center.

The radius R can be set as appropriate according to a situation to whichthe present exemplary embodiment is applied. For example, in the case ofobserving an indoor work, the radius R can be set to a distancecorresponding to 2 m. However, at this time, the control unit 121 refersto a history of coordinates stored in the node of the person serving asa processing target, and sets the value of the radius R, which is athreshold value, larger when determining that the person serving as aprocessing target is in motion than when determining that the personserving as a processing target is not in motion. Furthermore, at thistime, if collaborative workers are working side by side along the x-axisdirection on the horizontal plane image, the control unit 121 can use anellipse obtained by shortening a circle with the radius R in the y-axisdirection. Similarly, if collaborative workers are working side by sidealong the y-axis direction on the horizontal plane image, the controlunit 121 can use an ellipse obtained by shortening a circle with theradius R in the x-axis direction.

(Item 2). In a case where a person satisfying (Item 1) is present, thecontrol unit 121 determines whether that person is present in adetection area (work area frame).

In the present exemplary embodiment, the control unit 121 sets an areaspecified by the user operating, for example, the input device 130 as adetection area, in which a person is present. This enables preventing,for example, a non-worker who is moving on the opposite side of aglass-made wall or a person shown on, for example, a poster placed on adesk from being erroneously detected as a collaborative worker. However,setting of the detection area is not limited to inputting by the uservia, for example, the input device 130. The control unit 121 canautomatically set an area surrounded with a marker laid on the capturedimage as a detection area. The method for detection using a markerincludes, for example, performing pattern matching betweentwo-dimensional markers set at the four corners of a detection area andinformation on the above two-dimensional markers previously registeredwith the client apparatus 120 by performing raster scanning on thecaptured image.

In a case where all of the above-mentioned (Item 1) and (Item 2) aresatisfied, the control unit 121 determines that the person serving as aprocessing target is a collaborative worker (NO in step S1105), and thenproceeds to (Processing 3-1) (S1106). In other words, the control unit121 associates workers satisfying all of the above-mentioned (Item 1)and (Item 2) with each other and determines that the workers arecollaborative workers. Moreover, in a case where at least one of theabove-mentioned (Item 1) and (Item 2) is not satisfied, the control unit121 determines that the person serving as a processing target is anisolated worker (YES in step S1105), and then proceeds to (Processing3-2) (step S1109).

Here, processing for determining whether the person serving as aprocessing target is an isolated worker or a collaborative worker is notlimited to the above-described processing. For example, the control unit121 can be configured to associate persons with each other based on atleast one of positions of persons detected from the captured image,directions of faces of the persons, and distances between the persons.For example, the control unit 121 can be configured to associate personspresent within a predetermined range from a predetermined object, suchas a work table, in the captured image with each other, or can beconfigured to estimate directions (or eye lines) of faces of persons andto associate persons the directions of faces of whom intersect with eachother. Moreover, for example, the control unit 121 can be configured toassociate persons the distance between whom is within a predetermineddistance with each other. Then, the control unit 121 can be configuredto determine that persons each of whom is associated with another personare collaborative workers and a person who is not associated with anyother person is an isolated worker. Additionally, the control unit 121can be further configured to associate persons with each other based oncolors or shapes of uniforms, bibs (vest-like clothes), or caps ofpersons. For example, the control unit 121 can be configured toassociate persons the shapes of uniforms of whom are the same amongpersons the distance between whom is within a predetermined distancewith each other, and not to associate persons the shapes of uniforms ofwhom are different even among persons the distance between whom iswithin a predetermined distance with each other, or can make adetermination based on a plurality of conditions in such a manner thatpersons the distance between whom is within a predetermined distance andthe colors of bibs of whom are different are collaborative workers incombination and the other persons are isolated workers.

The determinations in the above-mentioned (Item 1) and (Item 2) aredescribed with reference to FIG. 20. In FIG. 20, ellipses 1201 to 1205represent results of converting the centers of the ellipses 401 to 405,which are person detection positions, into coordinates on the horizontalplane image. Moreover, a region 1206 represents a detection areaspecified by the user. Additionally, an ellipse 1200 represents anellipse defined by points x and y satisfying the above-mentioned formula(4).

In the above-mentioned formula (4), qx and qy denote the x coordinateand y coordinate, respectively, on the horizontal plane image of aperson serving as a processing target. In FIG. 20, qx and qy representthe x coordinate and y coordinate, respectively, of the ellipse 1202. Informula (4), a and b denote transformation parameters for an ellipse.For example, when a=1 and b=1, the ellipse becomes a circle with aradius of R and a center of (qx, qy). Moreover, when a=1 and b=½, theellipse becomes an ellipse with a length of R in the x-axis directionand a length of R/2 in the y-axis direction. In the present exemplaryembodiment, since collaborative workers are working side by side alongthe x-axis direction on the horizontal plane image, the values of thetransformation parameters are set to a relationship of “a>b”.

For example, suppose that the control unit 121 sets a person in theellipse 402, which is a person detection position, as a processingtarget during scanning of the list. At this time, the control unit 121performs determinations in the above-mentioned (Item 1) and (Item 2)with respect to the ellipse 1202 on the horizontal plane imagecorresponding to the ellipse 402, which is a person detection position.First, in (Item 1), the control unit 121 determines whether a detectedperson is present inside the ellipse 1200. In the situation illustratedin FIG. 20, since the coordinates of the ellipse 1201 on the horizontalplane image corresponding to the ellipse 401 for person detection areincluded in the ellipse 1200, the control unit 121 determines that therequired condition is satisfied. Next, in (Item 2), the control unit 121determines whether the coordinates of the ellipse 1201 on the horizontalplane image corresponding to the ellipse 401 for person detectioncausing satisfaction of the required condition of (Item 1) are includedin the region 1206. In the situation described in the present exemplaryembodiment, since the coordinates of the ellipse 1201 are included inthe region 1206, the control unit 121 determines that the requiredcondition is satisfied. Thus, since all of the requirements in (Item 1)and (Item 2) are satisfied, the control unit 121 determines that theperson in the ellipse 402 is a collaborative worker.

(Processing 3-1). In step S1106, the control unit 121 determines whetherthe last of the list is set as a processing target. If the processingtarget is not the last of the list (NO in step S1106), then in stepS1107, the control unit 121 sets a next person as a processing target,and then returns to (Processing 2). If the processing target is the lastof the list (YES in step S1106), the control unit 121 ends scanning onthe list, and then in step S1108, sets the cumulative number of framesin which an isolated worker is present to “0”.

(Processing 3-2). In step S1109, the control unit 121 increments thecumulative number of frames in which an isolated worker is present.

Next, in steps S1110 and S1111, the control unit 121 determines whetherto notify the user that an isolated worker is present, and, if thatcondition is satisfied, sends a notification to that effect.

With regard to, for example, display processing in issuing thenotification, the control unit 121 performs processing similar to thatin the first exemplary embodiment.

Furthermore, in the present exemplary embodiment, as a method fornotifying the user that an isolated worker is present, the control unit121 displays, on the display device 140, a user interface (UI)indicating that an isolated worker is present. For example, the controlunit 121 superimposes a double-line circle or superimposes a red circleat the position of an isolated worker on the captured image 300 or thehorizontal plane image 500. However, this is not limiting, but thecontrol unit 121 can, for example, send an e-mail to the user or canoutput a signal to an output terminal to issue an alert from a speaker.Next, in step S1112, the control unit 121 determines whether to repeatprocessing from acquisition of a captured image (step S1100). Forexample, the control unit 121 determines to end repetitive processing ina case where the acquisition of a captured image from the imagingapparatus 110 has become impossible or in a case where an instructionfor stopping the repetitive processing has been issued by the user.

Thus, according to the above-described fourth and fifth exemplaryembodiments, information indicating whether an isolated person ispresent or a plurality of collaborative persons is present in an imagecan be appropriately presented. When receiving a notification indicatingthat an isolated worker is present at a work site, the work observer canreadily check a video showing the isolated worker and thus can find aworker breaking a rule and call attention to the worker.

The present invention can also be implemented by supplying a programhaving one or more functions of the above-described exemplaryembodiments to a system or an apparatus via a network or a storagemedium. Then, one or more processors included in a computer of thesystem or the apparatus can read and execute the program. Moreover, thepresent invention can also be implemented by a circuit capable ofimplementing one or more functions (for example, application specificintegrated circuits (ASIC)).

While exemplary embodiments of the present invention have been describedin detail above, the present invention is not limited to such specificexemplary embodiments. The functional configuration of the imagingapparatus 110 can also be implemented in the imaging apparatus 110 as asoftware configuration that is implemented by the CPU 12 performingprocessing based on a program stored in the memory 13. Furthermore, apart or all of the functional configuration of the imaging apparatus 110can also be implemented in the imaging apparatus 110 as a hardwareconfiguration. Similarly, the functional configuration of the clientapparatus 120 can also be implemented in the client apparatus 120 by theCPU 22 performing processing based on a program stored in the memory 21.Additionally, a part or all of the functional configuration of theclient apparatus 120 can also be implemented in the client apparatus 120as a hardware configuration.

Furthermore, images of patterns to be superimposed on persons can beselected in any combination. More specifically, a collaborative workerand an isolated worker can be discriminated using respective differentcolors as well as using a solid-line ellipse and a double-line ellipse,and character strings indicating, for example, an isolated worker and acollaborative worker can be additionally displayed.

Moreover, the above-described hardware configuration of the clientapparatus or the imaging apparatus is merely an example, and, forexample, can include a plurality of CPUs, a plurality of memories, or aplurality of communication I/Fs.

According to the above-described exemplary embodiments, for example,information indicating whether an isolated person is present or aplurality of collaborative persons is present in an image can beappropriately presented.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random access memory (RAM), a read-only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. A control apparatus comprising: a detection unitconfigured to detect a person from an image which includes a pluralityof persons; an association unit configured to associate the personsincluded in the image with each other based on at least one of aposition of the person detected by the detection unit, directions offaces of the persons included in the image, and distances between thepersons included in the image; and an output control unit configured tocause an output unit to output information that is indicative of arelation of the detected person with respect to other persons includedin the image based on a result of association performed by theassociation unit.
 2. The control apparatus according to claim 1, whereinthe output control unit includes a display control unit configured tocause a display device to display the information that is based on theresult of association performed by the association unit, and configuredto cause the display device to perform display processing for enablingdiscriminating between a person who is associated with another person bythe association unit and a person who is not associated with any otherperson by the association unit.
 3. The control apparatus according toclaim 2, wherein the display processing for enabling discriminating isdisplay processing for enabling discriminating between a person who isassociated with another person by the association unit and a person whois not associated with any other person by the association unit using atleast one of a difference in type of a line, a difference in color of aline, a difference in type of a graphic, a difference in numericalvalue, and a difference in character string to be superimposed on eachperson.
 4. The control apparatus according to claim 2, furthercomprising an acquisition unit configured to acquire the image from animaging apparatus, wherein the detection unit detects a person from theimage acquired by the acquisition unit.
 5. The control apparatusaccording to claim 1, wherein the image is a moving image, and whereinthe output control unit includes a notification control unit configuredto cause a notification device to perform notification processingaccording to a period of time in which a person who is not associatedwith any other person by the association unit is contained in the movingimage.
 6. The control apparatus according to claim 5, wherein thenotification control unit causes the notification device to perform thenotification processing in a case where the period of time in which aperson who is not associated with any other person by the associationunit is contained in the moving image exceeds a threshold value.
 7. Thecontrol apparatus according to claim 5, wherein the notification deviceincludes a display device, and wherein the notification control unitcauses the notification processing to be performed by causing thedisplay device to perform display processing according the period oftime of a moving image in which a person who is not associated with anyother person by the association unit is contained.
 8. The controlapparatus according to claim 7, wherein the display processing isprocessing for superimposing, on the person included in the movingimage, an object indicating that the person is a person who is notassociated with any other person.
 9. The control apparatus according toclaim 5, further comprising an acquisition unit configured to acquirethe moving image from an imaging apparatus, wherein the detection unitdetects a person from the moving image acquired by the acquisition unit.10. The control apparatus according to claim 1, wherein the outputcontrol unit includes a display control unit configured to cause adisplay device to display the information that is based on the result ofassociation performed by the association unit, and wherein the displaycontrol unit causes the display device to display at least one ofinformation indicating that a person who is not associated with anyother person is contained in the image and information indicating that aperson who is not associated with any other person is not contained inthe image.
 11. The control apparatus according to claim 10, wherein theinformation distinguishably indicating that a person who is notassociated with any other person is contained in the image isinformation indicating a time period for which a person who is notassociated with any other person on a timeline is contained in theimage.
 12. The control apparatus according to claim 10, wherein theinformation distinguishably indicating that a person who is notassociated with any other person is not contained in the image isinformation indicating a time period for which a person who is notassociated with any other person on a timeline is not contained in theimage.
 13. The control apparatus according to claim 10, wherein theinformation distinguishably indicating that a person who is notassociated with any other person is contained in the image is at leastone of information indicating clock time of a moving image in which theperson is contained, information indicating a duration of the movingimage, and information indicating a location of the person.
 14. Thecontrol apparatus according to claim 10, further comprising anacquisition unit configured to acquire entrance and exit information onthe person from an entrance and exit management system, wherein thedisplay control unit causes the display device to further display theentrance and exit information.
 15. The control apparatus according toclaim 1, further comprising an acquisition unit configured to acquire animage from an imaging apparatus, wherein the detection unit detects aperson from the image acquired by the acquisition unit.
 16. The controlapparatus according to claim 1, wherein the detection unit detects aperson from the image based on a feature amount of the person.
 17. Thecontrol apparatus according to claim 1, wherein the association unitassociates persons with each other based on a feature amount of a thingworn by each person.
 18. The control apparatus according to claim 17,wherein the thing worn by each person is at least one of a uniform, abib, and a cap.
 19. A control method comprising: detecting a person froman image which includes a plurality of persons; associating the personsincluded in the image with each other based on at least one of aposition of the person detected from the image, directions of faces ofthe persons included in the image, and distances between the personsincluded in the image; and causing an output unit to output informationthat is indicative of a relation of the detected person with respect toother persons included in the image based on a result of theassociation.
 20. A non-transitory computer-readable storage mediumstoring computer-executable instructions that, when executed by acomputer, cause the computer to perform a method comprising: detecting aperson from an image which includes a plurality of persons; associatingthe persons included in the image with each other based on at least oneof a position of the person detected from the image, directions of facesof the persons included in the image, and distances between the personsincluded in the image; and causing an output unit to output informationthat is indicative of a relation of the detected person with respect toother persons included in the image based on a result of theassociation.